Attention is directed to a masters thesis by Nisha Checka entitled “A System for Tracking 15 and Characterizing Acoustic Impacts on Large Interactive Surfaces” (MIT, May, 2001) and J. D. Paradiso, K. Hsiao, J. Strickon, J. Lifton, and A. Adler, “Sensor Systems for Interactive Surfaces,” IBM Systems Journal 39, Nos. 3&4, 892-914 (2000), both of which are hereby incorporated by reference.
Large flat surfaces, such as glass windows, are common structures in everyday life. Because of a window's transparent nature, it is often used in showroom displays, or as a conduit through which one can view an area of a showroom. Without additional enabling circuitry, common glass windows are passive surfaces. While different products generally centered on the theme of “home automation” have inspired various interactive displays, these are usually hand-held, small or moderate-sized discrete electronic devices, such as touch screens or monitors that are embedded into walls, kiosks or tables. The use of large portions of the walls, floors, or windows themselves as interactive interfaces, except perhaps in niche applications such as those used for teleconferencing, is rare. As new technologies evolve, architectural surfaces are becoming more sensate, following trends and concepts in smart skins developed in the areas of structural control and aerospace research over the last decade.
Most of the commercial products that have been developed to track position across a large, responsive surface have been aimed at the hand-held digitizing tablet and “smart whiteboard’ markets, where handwriting from a writing instrument (such as a coded pen, or the styles used with the Palm Pilot™) is digitally captured. While many of these systems require contact or pressure to be applied against a sensitive surface and act as a large touch screen or trackpad, others detect the position of objects just above a board or tablet. The bulk of these devices are made to work with electronic sensing technology (usually nontransparent) within or beneath the active area. One interesting example of a recent, noncommercial sensing surface is the pixilated capacitive matrix devised by Post and collaborators at the MIT Media Lab for their sensor table developed for an installation at the Museum of Modern Art in New York. Although this technique can detect and track nearby bare hands through their capacitive loading, it does not scale easily in large areas and is generally opaque; therefore it is not suited to rear-projection applications. For smaller surfaces, transparent conductors such as indium-tin oxide (“ITO”) or conductive plastic can be used as in capacitive touchscreens, but extending such fine sampling or pixilated concepts to very large displays becomes complicated and expensive with existing technologies.
Most tracking systems for translucent or very large wallboards are the “squinting” type that look across from the edges of the display. Although inexpensive devices exist that use acoustic time-of-flight to a fixed receiver from active sonar pingers embedded in pens, several employ optical sensing, which enables simple, passive reflecting targets on the drawing objects to be easily detected in a sensitive plane defined by a scanning fan-collimated light source, such as generated by a scanned diode laser. For example, a pair of scanning lasers emanate from the two top corners of a board, identifying and tracking coded targets on pens and other objects approaching the whiteboard and intersecting the scanned plane. These sensors are unable to detect distance, thus planar position is determined by triangulating the two angular measurements. To avoid ambiguities in this triangulation, these systems generally allow only one object to be tracked at a time. Although such systems require coded targets, research systems have been designed to use a similar arrangement to track single fingers and bare hands. Light-Curtains, which use dense arrays of infrared light-emitting diodes (“IR LEDs”) that face corresponding receivers lining the perimeter of the screen, are commercially available and can track multiple hands, but because of poor scalability, become expensive for large displays. A variant on this approach is the Intrepid touch system, which uses an array of IR LEDs across the top of the display and two linear CCD arrays at the corners that look for reflections from the hands. Unfortunately, this technique can become problematic with large screens as it is expensive and illumination difficulties often persist.
Some smart wail hand-tracking systems use computer vision. The most straightforward versions of these use multiple cameras, squinting along the horizontal and vertical coordinates and triangulating. Although this approach is capable of providing a great deal of information potentially enabling hand gesture to be determined), it involves a considerable amount of sometimes fragile video processing to detect the hand, reject background light and clutter, and solve the image-correspondence problem for multiple hands.
Another technique is “chroma-key”, which is one that looks from the front of a screen. In this technique, the silhouette of the body is detected against a plain blue or bright screen, whereupon the hands are identified and tracked when not in front of the torso, similar to systems employed during weather broadcasts. Although the newscaster only gestures in front of a blue screen in the studio, the screen is replaced by the weather map in the broadcast video. For increased precision, lower ambiguity, higher speed, and the ability to work with variable background light or an image-bearing screen, many classic body-tracking systems have exploited active targets made from modulated IR LEDs that must be placed on or carried by the user.
Recent work by the MIT Media Laboratory's Responsive Environments Group on new user interface devices for interactive surfaces and large-scale public installations has led to the development of technologies such as an interactive wall which tracks hand positions with a low-cost scanning laser rangefinder and a smart window that locates finger taps using differential acoustic time-of-arrival. Such technologies, independently and in combination, turn an ordinary window into an input device, analogous to a touchscreen or mouse.